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Abstract 


The  continuous  transformation  of  the  US  military  into  an  integrated  and  a  dominant  “full  spectrum” 
capability  requires  a  comprehensive  and  evolutionary  system  model  (the  objective)  and  a  prescribed 
set  of  transformation  processes  (the  means)  whose  incremental  results  are  measurable  against 
operational  requirements.  Given  its  sheer  size  and  complexity,  this  military  system  must  be 
partitioned  into  logical  and  semi-independent  elements.  The  individual  and  ensemble  performance 
of  system  elements  must  be  objectively  measured,  and  characterized  by  specific  and  reusable 
metrics.  Taken  together,  the  system  model  and  the  classes  of  metrics  define  a  performance 
measurement  framework  (PMF).  This  paper  outlines  the  design  of  a  generalized  and  scalable 
performance  measurement  framework  for  the  Navy’s  FORCEnet  environment. 
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1,  Introduction 

Within  the  context  of  military  transformation,  end-to-end  performance  measurement  of  command 
and  control  (C2)  applications  in  “network  centric  warfare”  environments  represents  a  specific  and 
challenging  subtask.  Primary  challenges  derive  from  the  iterative  and  spiral  nature  of  long-term 
development  programs  and  the  inherent  performance  anomalies  found  in  distributed  large-scale 
enterprises.  [3]  These  challenges  notwithstanding,  a  practical  means  of  specifying  and  measuring  the 
behavior  of  DOD  C2  applications  is  required  for  1)  establishing  initial  system  and  application  design 
criteria,  2)  specifying  realistic  operational  baselines,  3)  defining  and  running  system  operations  and 
maintenance  diagnostic  processes,  and  4)  maintaining  reusable  and  scalable  sets  of  metrics  that 
effectively  support  historical  records  that  underlie  continuous  system  and  process  improvements. 

In  general,  command  and  control  environments  are  complex,  involving  chains  of  command  (i.e., 
accountability  hierarchies)  and  joint  forces  (i.e.,  distributed  peer-level  organizations)  that  evolve  in 
their  structure,  missions  and  objectives  over  time.  This  two  dimensional  context  provides  many 
sources  of  ambiguity  in  both  time  and  space.  In  order  to  define  performance  metrics  and  associated 
methods  for  making  consequential  and  reproducible  measurements,  a  scalable  and  extensible  model 
of  C2  enterprise  dynamics  is  required. 

The  DOD  transformation  objectives  are  represented  by  a  five  tier  pyramid  model  [5,  6,  8,  9,  and  10] 
with  a  grid-computing  foundation  and  “full  spectrum  dominance”  at  the  apex.  The  grid  hosts 
network-centric  warfare  capabilities  that,  at  the  intermediary  levels,  support  “information 
superiority”  and  “decision  superiority.”  Discussion  of  transformation  objectives  generally  refers  to 
levels  of  warfare.  [1,7] 
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Strategic  level  of  war  is  divided  into  two  sublevels:  strategie  national  (SN)  and  strategic  theater 
(ST).  At  this  level,  a  nation,  often  as  a  member  of  a  group  of  nations,  determines  national  or 
multinational  (alliance  or  coalition)  security  objectives  and  guidance,  and  develops  and  uses  national 
resources  to  accomplish  these  objectives.  Activities  at  this  level  establish  national  and  multinational 
military  objectives;  sequence  initiatives;  define  limits  and  assess  risks  for  the  use  of  military  and 
other  instruments  of  national  power;  develop  global  plans  or  theater  war  plans  to  achieve  these 
objectives;  and  provide  military  forces  and  other  capabilities  in  accordance  with  strategic  plans.  In 
the  [12],  this  level  of  war  is  divided  into  strategic -national  (DOD/Service/interagency)  and  strategic- 
theater  (combatant  command)  to  provide  clarity  and  focus  for  task  development  and  execution. 

Operational  level  of  war  is  where  campaigns  and  major  operations  are  planned,  conducted,  and 
sustained  to  accomplish  strategic  objectives  within  theaters  or  areas  of  operations.  Activities  at  this 
level  link  tactics  and  strategy  by  establishing  operational  objectives  needed  to  accomplish  the 
strategic  objectives,  sequencing  events  to  achieve  the  operational  objectives,  initiating  actions,  and 
applying  resources  to  bring  about  and  sustain  these  events.  These  activities  imply  a  broader 
dimension  of  time  or  space  than  do  tactics;  they  ensure  the  logistic  and  administrative  support  of 
tactical  forces  and  provide  the  means  by  which  tactical  successes  are  exploited  to  achieve  strategic 
objectives. 

Tactical  level  of  war  is  where  battles  and  engagements  are  planned  and  executed  to  accomplish 
military  objectives  assigned  to  tactical  units  or  task  forces.  Activities  at  this  level  focus  on  the 
ordered  arrangement  and  maneuver  of  combat  elements  in  relation  to  each  other  and  to  the  enemy  to 
achieve  combat  objectives. 

In  terms  of  levels  of  war,  the  DON  has  formulated  “joint  and  naval  performance  measures”  to 
support  its  transformation.  [12]  Performance  measures  previously  defined  for  joint  and  naval 
operations  contain  some  755  specific  performance  categories  -  218  at  the  strategic  level,  216  at  the 
operational  level,  and  321  tactical  level  measurements.  The  Navy’s  current  Sea  Trials  activity 
focuses  on  the  tactical  environment. 

The  FORCEnet  (Fn)  program  is  the  catalyst  for  transforming  the  US  Navy  into  a  “fully  netted  force” 
capable  of  engaging  “widely  distributed  combat  power  and  command  with  increased  awareness  and 
speed  as  an  integral  part  of  the  joint  team.”  [7]  The  roadmap  specifically  calls  for  three  major  phases 
of  development  and  deployment:  “Spiral  1”  (Figure  1)  from  FY02-07,  “Spiral  2”  from  FY07-10  and 
“Spiral  3  from  FYlO-20.  Spiral  1  comprises  a  set  of  requirements  and  processes  for  system 
experiments,  initiatives  and  acquisitions  providing  immediate  combat  capabilities.  Spiral  2 
comprises  activities  designed  to  meet  “compelling”  future  naval  challenges  in  the  FY07-10 
timeframe,  taking  into  consideration  the  FY04-10  science  and  technology  (S&T)  and  research  and 
development  (R&D)  roadmaps.  And  Spiral  3  is  aimed  at  creating  an  “enduring  asymmetric 
advantage”  for  the  Navy  of  FY 10-20,  coordinated  with  the  primary  platform  modernizations  and 
their  associated  requirements. 

A  performance  measurement  framework  that  is  applicable  to  these  three  development  periods 
requires  1)  a  generalized  model  of  distributed  real-time  command  and  control  systems,  2)  the 
FORCEnet  objectives  cast  within  this  distributed  C2  model,  and  3)  a  reusable  and  extensible  set  of 
strategic,  operational  and  tactical  performance  metrics  and  measurement  processes  that  capture  the 
essential  behaviors  of  deployed  C2  applications. 
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Figure  1  -  Spiral  Development  Process 


2,  C2  Systems  Model 

Major  Scott  F.  Murray,  US  Air  Force,  in  his  2002  article  Battle  Command,  Decisionmaking  and  the 
Battlefield  Panopticon  [13],  quotes  Biting  Morrison’s  conclusion  that  “the  primary  impediment  to 
exploiting  new  teehnologies  in  the  military  is  the  eultural  impact  of  organizational  change.”  This 
conclusion  is  not  unique  to  the  military.  It  is  endemic  to  medium  to  large  scale  bureaucracies, 
regardless  of  whether  they  exist  in  commercial, 
government  or  institutional  settings.  Given  the 
special  (e.g.,  large  and  lethal)  nature  of  the  military 
command  and  control  environment,  the  point  is 
especially  relevant.  As  a  consequence,  the  nature  of 
organization  and  the  process  of  command  and  control 
within  that  organization  are  eritical  to  the  success  of 
transformation  activities. 

Figure  2  illustrates  the  PMF  C2  systems  model.  The 
model  defines  five  echelons  of  control,  as  defined  in 
cybernetics  and  developed  from  models  of  human 
neuro-anatomy. [3]  At  the  highest  level,  echelon  5 
(E5)  defines  a  system’s  supervisory  control,  its 
identity,  purpose  and  vision  (doctrine)  -  the  basis  for 
strategy  and  aetion  -  where  the  context  for  its 
perceptions  (belief  system)  and  behaviors 
(prerogatives)  takes  form.  For  the  Department  of  the 

Navy,  E5  represents  the  CNO.  Echelon  4  (E4)  ^  ^ 

.  +  ^  1  ^  1  Figure  2  -  Echelons  of  Control 

provides  innovation,  development  and  planning  “ 
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functions,  whereby  the  state  of  the  external  and  internal  worlds  are  rationalized  and  incorporated  into 
the  forward-looking  strategies  for  guiding  the  system’s  adaptation  and  evolution.  Applied  to  the  US 
Navy,  E4  is  represented  in  part  by  the  CNO  N6  direetorate.  Echelon  3  (E3)  is  operationally  foeused, 
as  through  the  Navy’s  XOs,  providing  the  function  of  supervision  through  audit  (i.e., 
parasympathetic,  E3*)  and  delegation  (i.e.,  sympathetic  or  autonomic  control)  of  lower  level 
processes.  Echelon  2  (E2)  provides  the  per-process  regulatory  controls,  providing  synchronization 
and  conflict  resolution  (i.e.,  local  sympathetic)  behavior  of  the  individual  processes  that  share 
resources  and  missions.  And,  where  the  real  work  of  the  enterprise  is  carried  out,  echelon  1  (El) 
contains  the  logic  and  machinery  of  the  individual  processes,  along  with  their  interfaces  to  the 
embedded  lower  level  services  on  whieh  they  in  turn  depend. 

Viable  systems  are  embedded  in  operational  environments,  where  external  realities  dictate 
conditions  for  action  and  survival.  This  external  environment  (Eigure  3)  defines  the  dynamic 
political,  social  and  battle  contexts  within  whieh  the  viability  of  the  system  -  and  hence  the  critical 
success  faetors  (measures  of  performance)  that  govern  good  decision  making  -  takes  plaee.  Within 
these  external  contexts  a  system  may  be  proaetive,  and  it  must  certainly  be  reactive. 
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Figure  3  -  Operational  Contexts 
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A  system’s  viability,  therefore,  must  take  into 
consideration  measures  of  both  internal  states  and 
external  state.  The  timely  correlation  of  these 
measurements,  and  the  world  view  this  correlation 
provides,  defines  the  basis  for  analysis  and 
subsequent  action.  Clearly,  correlated  measurements 
that  arrive  too  early  or  too  late  have  less  value  to  the 
E3,  E4  and  E5  control  functions  than  correlated 
measures  that  arrive  “just  in  time”  (JIT).  Viable  C2 
systems  must  therefore  be  sensitive  to  the  value  of 
time  critical  measures  -  in  short,  a  C2  system  is  a 
real-time  distributed  system  -  in  all  of  its  technical 
meaning. 

As  diagrammed  in  Figure  3  each  El  “production 
process,”  at  the  next  lower  level,  may  be  recursively 
represented  by  a  control  structure  similar  to  Figure  2. 
Applying  the  model  to  the  DOD,  the  top  level  system 
model  might  be  drawn  as  in  Figure  4,  where  the  core 
El  processes  include  the  CJCS/JCS/Staff,  the  Joint 
Combat  Commanders,  and  the  Navy,  Marine  Corps, 
Air  Force,  and  Army  forces.  E5  here  is  the  OSD,  E4 
the  ASD  and  its  technical  support  directorates,  and  so 
on. 


E5 


For  the  purpose  of  developing  a  unified  and  scalable  Figure  4  -  SECDEF  Control  Domains 
performance  measurement  framework,  the  system 

model  is  applied  recursively  to  the  six  El  processes  of  Figure  4,  respectively,  to  identify  then- 
individual  operational  structures.  The  number  of  recursions  is  equal  to  the  number  of  desired  levels 
of  C2  behavior  one  wishes  to  observe  (measure).  In  practice,  six  levels  have  been  shown  sufficient 
to  identify  the  essential  behavior  of  large  enterprises.  In  the  commercial  world  [3]  these  levels 
translate  into  multi-domestic  Corporations  (F5),  Business  Areas  (aka  “Division,”  F4),  Business 
Units  (F3),  Production  Plants  (F2),  Production  Processes  (El),  and  Production  Units  (aka 
“Intelligent  Devices,”  EO).  The  military  equivalent  has  yet  to  be  firmly  established  and  exploited, 
but  for  the  purposes  of  this  work  we  believe  that  the  six-level  cybernetic  model  is  also  adequate. 
Clearly,  the  PMF  requires  some  form  of  system  reference  model,  one  that  specifies  an  appropriate 
level  of  functional  detail,  but  one  that  does  not  cause  such  complexity  that  it  obscures  the  value  of 
performance  measurement.  [14] 

Figure  5  expands  Figure  4’s  El  Navy  process  into  four  additional  levels.  This  particular  “view”  of 
the  Navy’s  organization  begins  with  the  SECDEF  at  the  top  and  ends  at  the  lowest  level  with,  in  this 
case,  a  particular  naval  vessel  within  a  battle  group.  The  nesting  could  go  down  to  the  level  of  a 
piece  of  automated  equipment  on  the  vessel  (e.g.,  a  semi-automatic  “intelligent  weapon”),  and  up  to 
the  Commander  in  Chief  (POTUS).  In  the  PMF  the  number  of  levels  shall  be  determined  by  the 
focus  of  the  measurement  program  and  the  desired  granularity,  accuracy  and  inter-relationships 
among  the  metrics  required  for  analyzing  effectiveness  of  selected  C2  policies  and  mechanisms.  In 
this  regard,  the  model  is  highly  scalable. 
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:  Carrier 


Figure  5  -  Navy  C2  Command  Levels 

Figure  5  follows  a  given  path  (or  “thread”)  of  operational  command  authority  appropriate  for 
defining  the  performance  of  a  battle  group  within  fleet  operations.  In  this  situation,  the  PMF  might 
be  configured  to  provide  real-time  metrics  for  characterizing  the  behavior  of  a  carrier  within  a  battle 
group  as  that  group  operates  within  a  time  critical  fleet  asset  management  scenario.  What  is  not 
depicted  here,  and  which  may  indeed  be  more  critical,  is  the  behavior  of  that  same  carrier  within  the 
purview  of  the  Combat  Commanders  responsible  for  a  tactical  military  action  requiring  the  carrier’s 
assets  (e.g.,  assault  support).  Such  a  view  would  require  the  PMF  to  change  “focus,”  moving  to  a 
different  aspect  of  the  command  structure,  this  time  with  CNO  replaced  by  the  Combat 
Commander(s)  and  the  FLT  replaced  by  the  Joint  Force  Commands. 

The  PMF  must  therefore  provide  a  means  of  (re)focusing  the  measurement  system  dynamically, 
much  as  an  astronomer  repositions  and  refocuses  a  telescope.  In  fact,  the  relative  performance  of 
any  given  set  of  Naval  assets  may  well  be  the  convolution  or  ensemble  average  of  the  performance 
of  that  element  as  seen  through  several  such  views,  with  appropriate  weighting  being  dependent  on 
battlespace  situation,  time  of  day,  BDA,  etc.  Important  design  criteria  for  the  PMF  include  this  user 


6  of  10 


interface,  its  controls,  its  fidelity,  and  its  ability  to  define  appropriate  foci  for  time-critical 
measurements  and  measurement  processes.  The  PMF  user  interface  is  the  subject  of  a  specific  R&D 
proposal  presented  elsewhere. 

3,  Performance  Metrics 

As  noted,  the  PMF  is  required  to  be  a  real-time  system,  meaning  that  metrics  -  whether  strategic, 
operational  or  tactical  -  must  carry  along  with  their  acquired  “values”  the  time  the  measurements 
were  made  (i.e.,  a  time-stamp),  and  some  measure  of  the  “quality”  of  those  values  at  that  time.  For 
reasons  of  computing  causal  relationships  among  measurement  events,  time  stamps  must  be  in 
absolute  time  (e.g.,  UTC/GPS).  The  quality  parameter(s)  associated  with  this  measurement  must 
define  the  status  of  the  sensors  in  terms  of  their  ability  to  render  specific  degrees  (ranges)  of 
accuracy  and  precision  -  the  fidelity  issue. 

A  system  is  real-time  to  the  degree  that  its  correct  behavior  requires  adherence  to  certain 
completion-time  semantics  (i.e.,  its  ability  to  meet  time  constraints  and  the  predictability  of  meeting 
deadlines).  [1 1]  A  distributed  system  is  real-time  to  the  degree  that  transnode  computations  (aka, 
distributed  threads)  have  predictable  timing  as  they  flow  through  the  interconnecting  grid.  This 
requires  that  network  latencies  be  bounded  and  known  a  priori  by  their  probability  distribution 
functions.  Parenthetically,  it  is  noted  that  the  current  level  of  effort  being  expended  in  the  NMCI 
program  [15]  does  not  formally  address  the  complex  technical  issues  of  end-to-end  timeliness  in  the 
naval  intranet,  the  associated  service  level  agreements  (SLA)  supporting  such  qualities-of-service 
(QoS),  nor  the  “costs”  of  missing  the  timeliness  “guarantees”  that  C2  applications  might  incur  in  the 
face  of  failed,  late,  or  corrupted  “transnode  threads.” 

The  quality  of  any  given  C2  application,  and  the  ultimate  utility  of  the  PMF,  is  dependent  on  the 
capability  of  the  sensors  that  make  the  measurements  at  each  level,  the  timeliness  of  the  transport 
system  that  delivers  the  payload  containing  the  measurement,  the  ability  of  the  receiving  sensory 
perception  process  that  filters  or  correlates  the  measurement,  and  so  on. 

The  PMF  defines  a  measurement  as: 

measurement  id  :=  {command  level, 

control  echelon, 
sensor_id, 
sensor_value, 
sensor  timestamp, 
sensor_quality } 

For  each  critical  metric  defined  in  [2]  and  [12]  the  PMF  assigns  and  registers  unique  measurement 
identifications  (MID). 

Figure  6  depicts  a  three  dimensional  measurement  id  space,  bounded  in  depth  by  six  levels  of 
control,  vertically  by  five  echelons  of  system  regulation,  and  horizontally  by  the  three  classes  of 
measurements  -  strategic,  operational  and  tactical  -  over  750  in  all.  The  figure  also  depicts  two 
representative  measurement  SI  and  S2. 
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An  important  goal  of  the  PMF 
is  to  answer  questions  sueh  as 
“what  effects  does  the 
performance  of  a  L2 
operational  commander  (i.e., 

C2  application  set  SI)  have  on 
the  performance  of  a  tactical 
commander  at  LI  (i.e.,  C2 
application  set  S2)?”  While 
the  figure  does  not  answer  such 
questions,  it  provides  a  degree 
of  clarity  as  to  the  dimensions 
(e.g.,  vector  lengths)  of  the 
issues.  It  also  introduces  the 
notion  of  “measurement  sets” 
and  their  importance  in 
quantifying  the  essential 
performance  parameters  for 
given  C2  situations. 


Figure  6  -  3D  Measurement  ID  Space 


The  current  phase  of  the  PMF 
development  effort  focuses  on 
the  lowest  (tactical)  group, 
albeit  the  one  with  the  highest 

data  rate  and  number  of  measurements  categories  -  over  300  are  defined  in  [12]  for  this  group. 

Each  measurement  category  in  this  group  contains  an  average  of  over  5  metrics  (e.g.,  speed, 
distance,  percentage,  completions,  events,  etc.).  Given  the  model  chosen,  the  measurement  space  for 
this  tactical  group  contains  well  over  10,000  possible  metrics  (300  x  5  x  6  x  5),  the  relevant  subsets 
being  functions  of  the  level  of  granularity  (specificity)  required  and  the  focus  of  C2  performance 
monitoring  objectives. 


Any  data  set  of  that  size  would  be  difficult  to  interpret  if  it  were  time  invariant.  By  the  nature  of 
evolving  battle  conditions  and  the  stochastic  character  of  network  transport  (e.g.,  the  QoS  issues 
mentioned  above  in  the  NMCI  alone),  the  measurement  space  is  clearly  time-varying.  In  fact,  it 
likely  to  be  discovered  that  the  space  contains  processes  that  are  neither  stationary  nor  ergodic  -  an 
analysis  that  will  have  to  await  the  deployment  of  the  PMF  measurement  “probes”  and  the 
evaluation  of  the  results  of  several  measurement  sets  from  different  applications  and  over  different 
time  periods. 


As  partial  compensation  for  the  numerical  complexity  and  end-to-end  performance  uncertainties 
expected  in  measurement  statistics,  the  PMF  defines  several  generalized  basis  measures  that  apply 
equally  well  across  all  command  levels.  These  measures  are  “unit-less,”  with  their  relationships 
defined  below  in  Figure  7.  These  “basis  metrics”  are  derived  from  the  operation  of  the  echelon 
controllers. 
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Figure  7  -  Generalized  “Basis”  Metrics 


1 .  Potential . The  design  potential  or  constraints  of  a  given  C2  function,  determined  by  its 

architecture. 

2.  Capability . The  [allocated  or  funded]  level  of  capacity  of  a  given  C2  function. 

3.  Actuality . The  current  level  of  actual  measured  performance  of  a  given  C2  function. 

4.  Latency . The  ratio  of  capability  to  potential  -  representing  the  [as  yet  unallocated] 

capacity  available  within  a  given  design. 

5.  Potential . The  ratio  of  actuality  to  capability  -  the  [remaining  or  allocatable]  “headroom” 

available  without  altering  the  design  potential  (i.e.,  allocatable  or  fundable 
capacity). 

6.  Performance . The  ratio  of  actuality  to  potential  -  the  present  efficiency  of  a  system  relative  to 

its  design. 


Given  the  recursive  nature  of  the  system  model,  these  generalized  measures  are  produced  at  each 
level  of  C2  hierarchy,  providing  a  means  of  comparing  and  accumulating  the  performance  of  the 
embedded  systems  and  their  associated  effects  on  the  performance  of  the  C2  environments  within 
which  they  operate. 

4,  Performance  Measurements 

The  process  of  taking  and  analyzing  performance  measurements  is  the  subject  of  a  separate  paper. 
For  the  purpose  of  this  introduction,  the  PMF  methodology  recognizes  that  commands  from  Level  0 
“intelligenf  ’  devices  through  Level  5  senior  officers  all  produce  meaningful  and  relevant  measures 
of  performance.  As  such,  functioning  elements  at  each  level  require  sensors  that  collect  and  transmit 
performance  data.  These  sensors  (servers)  and  the  effectors  they  feed  (clients)  are  in  addition  to 
those  sensors  and  effectors  that  carry  out  the  war  fighting  mission  itself  The  PMF  is  therefore  a 
“metasystem”  within  which  the  combat  systems  operate  and  are  evaluated.  The  measurement 
processes  must  therefore  be  able  to  “monitor”  both  classes  of  end-systems,  both  mission  assets  and 
mission  infrastructure. 
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The  PMF  recognizes  both  classes  of  end  system  in  the  grid-based  environment  defined  by 
FORCEnet  and  its  containing  Unified  Command  System  (UCS)  [7,  8,  and  16].  The  PMF 
mechanisms  planned  for  identifying  and  measuring  performance  are  based  on  the  concept  of  both 
synchronous  and  asynchronous  probes  (aka,  threads  or  intelligent  agents)  that  traverse  the  grid  to 
evaluate  sensor  state  (quality),  make  measurements  (value),  and  time-stamp  the  events  at  end-system 
elements  engaged  in  C2  applications  of  interest.  In  doing  so  they  also  assess  the  end-to-end  transit 
times  that  contribute  to  overall  performance.  In  addition,  PMF  measurement  agents  are  dispatched 
to  various  network  nodes  within  the  grid  environment  to  act  as  diagnostic  servers.  The  particular 
design,  deployment  and  management  of  these  probes  and  the  databases  that  result  from  their 
operation  will  be  the  subject  of  a  subsequent  paper. 

5.  Conclusions 

We  have  presented  the  general  characteristics  of  the  design  and  specification  of  a  performance 
measure  framework  for  C2  application  in  a  network-centric  military  environment.  The  PMF  is 
based  on  a  recursive  system  model  that  recognizes  five  control  elements.  The  recursion  depth  is 
determined  by  the  number  of  command  levels  to  be  considered.  The  measurements  to  be  used  by  the 
PMF  include  those  strategic,  operational  and  tactical  metrics  identified  by  the  DON  in  its  assessment 
of  core  requirements,  and  a  set  of  generalized  measures  that  apply  to  all  command  levels. 
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